System and method for detecting polyps from learned boundaries

ABSTRACT

A system and method for automated polyp detection in optical colonoscopy images is provided. In one embodiment, the system and method for polyp detection is based on an observation that image appearance around polyp boundaries differs from that of other boundaries in colonoscopy images. To reduce vulnerability against misleading objects, the image processing method localizes polyps by detecting polyp boundaries, while filtering out irrelevant boundaries, with a generative-discriminative model. To filter out irrelevant boundaries, a boundary removal mechanism is provided that captures changes in image appearance across polyp boundaries. Thus, in this embodiment the boundary removal mechanism is minimally affected by texture visibility limitations. In addition, a vote accumulation scheme is applied that enables polyp localization from fragmented edge segmentation maps without identification of whole polyp boundaries.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Patent Application No.61/983,868 filed Apr. 24, 2014.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

N/A

BACKGROUND

The subject matter described herein relates to systems and methods forprocessing optical images, and, more particularly, to automaticdetection of polyps in optical images.

Colorectal cancer (CRC) is the second highest cause of cancer-relateddeaths in the United States with 50,830 estimated deaths in 2013. Morethan 80% of CRC cases arise from adenomatous polyps, which areprecancerous abnormal growths of the colon wall. The preferred screeningmethod for polyp detection and removal is an optical colonoscopy (OC)procedure, during which a colonoscopist meticulously examines the colonwall using a tiny camera that is inserted and guided through the colon.The goal of an OC is to detect and remove colorectal polyps, which maybe precursors to CRC. Thus, it has been shown that timely removal ofpolyps can significantly reduce the mortality of CRC.

However, polyp detection with OC remains a challenging task and, asevidenced by several clinical studies, a significant portion of flat andpedunculated polyps remain undetected during colon screening with OC.High polyp detection rate requires a high level of attentiveness,alertness, and sensitivity to visual characteristics of polyps fromcolonoscopists and such qualities may only be procured after years ofpractice and experience. It is therefore important to reduce polypmiss-rate as it decreases the incidence and mortality of CRC.

Computer-aided polyp detection has recently been considered as a toolfor reducing polyp miss-rate. For example, during an OC procedure,regions with suspected polyps can be highlighted for furtherexamination. Existing approaches for polyp detection primarily rely onthe shape or texture of polyps. However, shape information issusceptible to partial and fragmented image segmentation and can misleada detector towards irrelevant objects in the complex endoluminal scene.Texture may also be unreliable because its visibility depends oncamera-polyp distance. Thus, the texture of a polyp becomes fullyvisible only when the camera captures close shots of the surface of apolyp. This condition is often met when polyps have already beendetected by operators. On the other hand, shape information cannot beconsidered as a reliable measure because polyps appear in a variety offorms ranging from sessile to peduncular shapes. Therefore,texture-based and shape-based polyp detectors offer limited practicalvalue.

Consequently, considering such limitations of previous technologicalapproaches, it would be desirable to have a system and method foraccurate and reliable polyp detection in optical colonoscopy images thatis shape-based and can compensate for the concomitant drawbacks ofshape-based detection.

SUMMARY

In accordance with one aspect, a system for automated polyp detection inoptical colonoscopy images is disclosed. The system includes an inputconfigured to acquire a series of optical images, a processor, and amemory. The memory contains instructions that, when executed by theprocessor, causes the processor to perform a process on the series ofoptical images. The processor applies a color filter to create aplurality of color filtered images for each optical image of the seriesof optical images, and locates a series of edge pixels on at least oneof the plurality of color filtered images. The process then obtains aplurality of oriented image patches corresponding to each of the seriesof the edge pixels. The plurality of oriented image patches arerepresented by an intensity signal characterized by at least one ofrotation invariance or illumination invariance. An edge normal for eachedge pixel is then estimated. At least one classification system isconstructed corresponding to the plurality of color filtered images. Theat least one classification system is configured to enhance low levelfeatures of the plurality of color filtered images prior toclassification, and appearance features are generated from a series offeature vectors selected from the classification system. An edgeclassification threshold analysis is then performed on each of theplurality of oriented image patches. Based on the edge classificationthreshold analysis, the processor determines that a given image patch isconsistent with an edge threshold. A report is generated indicatingpotential polyps with a vote accumulation greater than a threshold. Thevote accumulation includes a probabilistic output for each of thepotential polyps in the absence of a predefined parametric model.

In accordance with another aspect, a method for automated polypdetection in optical colonoscopy images is disclosed. The methodincludes applying a color filter to create a plurality of color filteredimages for each of a plurality of optical images. A series of edgepixels are located on at least one of the plurality of color filteredimages, and a plurality of oriented image patches are obtainedcorresponding to each of the series of the edge pixels. The plurality oforiented image patches are represented by an intensity signalcharacterized by at least one of rotation invariance or illuminationinvariance. An edge normal is estimated for each edge pixel, and atleast one classification system is constructed corresponding to theplurality of color filtered images. The at least one classificationsystem is configured to enhance low level features of the plurality ofcolor filtered images prior to classification. Appearance features aregenerated from a series of feature vectors selected from the at leastone classification system, and an edge classification threshold analysisis performed on each of the plurality of oriented image patches. Basedon the edge classification threshold analysis, a given image patch isdetermined to be consistent with an edge threshold. A report isgenerated indicating potential polyps with a vote accumulation greaterthan a threshold. The vote accumulation includes a probabilistic outputfor each of the potential polyps in the absence of a predefinedparametric model.

The foregoing and other aspects and advantages of the disclosure willappear from the following description. In the description, reference ismade to the accompanying drawings which form a part hereof, and in whichthere is shown by way of illustration one embodiment. Such embodimentdoes not necessarily represent the full scope of the disclosure,however, and reference is made therefore to the claims and herein forinterpreting the scope of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an exemplary system for polypdetection in optical colonoscopy images in accordance with the presentdisclosure.

FIG. 2 is a flowchart setting forth steps of an exemplary method ofoperating an automated polyp detection system in accordance with thepresent disclosure.

FIG. 3 is a schematic illustration of a process for automated polypdetection in optical colonoscopy images in accordance with the presentdisclosure.

FIG. 4 is a flowchart setting forth steps of an exemplary method forpolyp detection in accordance with the present disclosure.

FIG. 5A is an illustration showing a ground truth of a polyp andcorresponding upright and oriented image patches along the edgedirection.

FIG. 5B is an illustration showing estimated edge normals using agradient-based approach and a ball tensor approach for the regionhighlighted in FIG. 5A.

FIG. 6 is a series of six eigen images selected from a two-stageclassification system.

FIG. 7A is a geometric illustration of a voting scheme for an edge pixellying at 135 degrees.

FIG. 7B is an illustration of an original voting map for parallel andcircular edges.

FIG. 7C is an illustration of a modified voting map for parallel andcircular edges.

FIG. 7D is an illustration of a voting map for a polyp showing searchradii for a subset of radial rays.

FIG. 8A is an illustration of another voting map having isocontoursgenerated for a synthetic shape.

FIG. 8B is an illustration of a representative isocontour Φ of thevoting map of FIG. 8A that is used to determine the shape and width of acorresponding narrow band.

FIG. 9A is a graph comparing resultant ROC curves of an image descriptorusing three selected coefficients and ROC curves obtained from otherwidely-used methods.

FIG. 9B is an illustration of a discrimination map showing adiscrimination power of each feature extracted by the image descriptoracross a series of images.

FIG. 9C is an illustration of the average appearance of a polyp boundaryrevealing informative features.

FIG. 9D a graph showing an ROC curve for a polyp detection systemaccording to the present disclosure.

FIG. 10 is a graph showing the effect of Gaussian Smoothing on thesensitivity of a Canny edge detector and the sensitivity of a votingscheme.

FIG. 11 shows true polyp detection results on voting maps superimposedon colonoscopy images.

FIG. 12 is a precision recall of polyp detection table comparing variouspolyp detection systems.

FIG. 13 a precision recall of polyp detection table comparing variouspolyp detection systems.

FIG. 14 shows true polyp detection results on voting maps superimposedon colonoscopy images.

FIG. 15 is a graph comparing resultant ROC curves of a two-stageclassification scheme that outperforms a one-stage classificationscheme.

DETAILED DESCRIPTION

The present disclosure describes embodiments that overcome theaforementioned drawbacks by providing a system and method for polypdetection based on an observation that image appearance around the polypboundaries differs from that of other boundaries in colonoscopy images.To reduce vulnerability against misleading objects, the imagingprocessing method localizes polyps by detecting polyp boundaries, whilefiltering out irrelevant boundaries, with classification and featureextraction methods. To filter out irrelevant boundaries, a boundaryremoval mechanism is provided that captures changes in image appearanceacross polyp boundaries. Thus, the boundary removal mechanism isminimally affected by texture visibility limitations. In addition, thepresent disclosure describes embodiments that overcome the challengesposed by partial polyp segmentation by applying a vote accumulationscheme that enables polyp localization from fragmented edge segmentationmaps without requiring perfect identification of whole polyp boundariesand knowledge about shapes and size of polyps. Therefore, the systemsand methods as described herein assist both experienced, andinexperienced, colonoscopists with accurately detecting and locatingpolyps.

In addition, the present disclosure describes embodiments that overcomethe aforementioned drawbacks by providing a system and method for polypdetection that combines image context with shape information to minimizethe misleading effect of irrelevant objects with polyp-like boundaries.Given an input image, the present method begins with collecting a crudeset of boundary pixels that are refined by a patch descriptor andclassification scheme, before feeding a voting scheme for polyplocalization. The patch descriptor quickly and efficiently characterizesimage appearance across object boundaries, and is both rotationinvariant and robust against linear illumination changes. A two-stageclassification framework is also provided that is able to enhance lowlevel image features prior to classification. Unlike traditional imageclassification where a single patch undergoes the processing pipeline,the present system fuses the information extracted from a pair ofpatches for more accurate edge classification. In addition, a voteaccumulation scheme that robustly detects objects with curvy boundariesin fragmented edge maps is provided. The voting scheme produces aprobabilistic output for each polyp candidate, but does not require anypredefined parametric model of polyps (e.g., circle and ellipse).

Thus, the disclosed polyp detection system and method are based on twokey observations. First, polyps, irrespective of their morphology,feature a curvy segment in their boundaries. The polyp detection systemuses this property to localize polyps by detecting objects with curvyboundaries. Second, image appearance across the polyp boundaries ishighly distinct from that of vessels, lumen, and specular reflections.Thus, present patch descriptor and classification scheme is able todistinguish polyp boundaries from the boundaries of other colonicobjects, producing a refined edge map for the vote accumulation scheme.

The methodology of the present disclosure is based on image appearancevariation between polyps and their surrounding tissue. The rationaletakes into account that local patterns of color variation across theboundary of polyps differ from the patterns of color variation thatoccur across the boundary of folds, lumen, and vessels.

Turning to FIG. 1, a block diagram is shown of an exemplary polypdetection system 100, which facilitates the detection of polyps onoptical images gathered from a subject 102. The polyp detection system100 generally may include image acquisition hardware 104, a processor106, an input 108, an output 110, a memory 112, and any device forreading computer-readable media (not shown). The polyp detection system100 may be, for example, a workstation, a notebook computer, a personaldigital assistant (PDA), a multimedia device, a network server, amainframe or any other general-purpose or application-specific computingdevice. The polyp detection system 100 may operate autonomously orsemi-autonomously, or may read executable software instructions from acomputer-readable medium (such as a hard drive, a CD-ROM, flash memoryand the like), or may receive instructions from a user, or any anothersource logically connected to a computer or a device, such as anothernetworked computer or server. In one embodiment, the polyp detectionsystem 100 is configured to acquire and analyze optical image data inreal-time from a live feed, while a medical procedure is being performedon a subject 102, such as a colonoscopy, and is also configured toretrieve and analyze optical image data already acquired and stored inany image data storage location.

The image acquisition hardware 104 may be designed to acquire opticalimage data continuously or intermittently, for example, during a medicalprocedure, such as a colonoscopy, and relay optical image data forprocessing. The image acquisition hardware 104 may require operatordirection, input or feedback, or may be designed to operateautonomously.

The processor 106 may be configured to process optical image data,including image data obtained during a medical procedure, such as acolonoscopy. In one embodiment, the processor 106 may be designed toprocess optical images, generated from optical image data, by applying aplurality of color filters. One non-limiting example of a plurality offilters may include a red (R), green (G) and blue (B) filter, oftenreferred to as an RGB filter. Within this example, anhue-saturation-lightness (HSL) or hue-saturation-value (HSV) coordinaterepresentation of the RGB model may be used. Other non-limiting examplesof color maps include La*b* (or Lab color space). In addition, it ispossible to use more than one color map, for instance, RGB+La*b*.Regardless of the filter, map, color space, particular combination offilters, maps, or color spaces, the present disclosure provides a systemand method for polyp detection that leverages the appearance of colorvariation between polyps and surrounding tissues.

The input 108 may take any suitable shape or form, as desired, foroperation of the polyp detection system 100, including the ability forselecting, entering or otherwise specifying parameters consistent withdetecting polyps of a requisite or desired size or shape.

The output 110 may take any suitable shape or form, as desired, and mayinclude a visual and/or audio system, configured for displaying, forexample, acquired optical images as a result of a medical procedure,such as a colonoscopy, and also configured, for example, to highlightand/or alert an operator of the polyp detection system 100 uponidentification of a polyp with the requisite or desired features.

The memory 112 may contain software 114 and data 116, and may beconfigured for storage and retrieval of image processing information anddata to be processed by the processor 106. In one embodiment, thesoftware 114 includes instructions directed to performing optical imageprocessing for polyp detection. In another embodiment, the data 116includes optical image data.

Turning to FIG. 2, a process 200 flowchart is shown illustrative of anexemplary method of operation for a polyp detection system 100. Theprocess begins at process block 202, wherein optical image datagenerated from a live feed, or retrieved from any image data storagelocation, such as the memory 112, is acquired in the form of an opticalimage by the polyp detection system 100. At process block 204, a polypdetection is performed on the optical image.

Illustrating the general steps associated with performing the polypdetection of process block 204, is a flow diagram shown in FIG. 3. Thepolyp detection process 300 begins with an optical image 302, whichundergoes an edge detection and edge direction estimation at processblock 304, wherein a crude set of edge pixels are generated to form acorresponding crude edge map using any suitable edge detector, such as aCanny edge detector. In some embodiments, the Canny edge detector isreplaced by a raster scan, however this approach may yield increasedcomputational costs. Thus, because the primary goal is to detect polypedges, the edge detector may be utilized to limit the search space to asmall subset of image pixels.

Next, at process block 306, image patches 305 of polyp boundaries arecaptured by extracting oriented sub-images 307 along the edge normalsthat correspond to the R, G, and B color channels. The featureextraction method is then applied on the oriented sub-images 307 togenerate appearance features. Next, at process block 308, the crude edgemap is refined by means of a classification scheme that operates on theextracted features. More specifically, a two-stage classification systemis used to filter out irrelevant, non-polyp edges (e.g., those edgeslying on folds and vessels), while retaining edges around and on thepolyp boundary. By filtering out the irrelevant boundaries, the changein image appearance across the boundary of polyps is revealed, whichdiffers from what is observed across other boundaries. Thus, imagepatches 305 that are extracted from polyp boundaries may be referred toas “positive patches” and the remaining image patches as “negativepatches.” Similarly, the edges that lie on the boundary of polyps may bereferred to as “polyp edges” and the remaining edges may be referred toas “non-polyp edges.” Once the crude edge map is refined, at the nextprocess block 310, a vote accumulation scheme is applied to the refinedmap to localize polyps.

FIG. 4 illustrates an exemplary method for polyp detection. Referring toFIG. 4, the polyp detection process 400 begins at process block 402,where a number of pre-processing steps may be applied to the opticalimage 302, including Gaussian smoothing and/or color filtering. HeavyGaussian smoothing at process block 402 may remove a large portion ofedge pixels including those on the boundaries of polyps. By contrast,slight smoothing may include both prominent and irrelevant edges, whichnot only may pose a large computational burden on the later stages, butalso may increase the chance of producing false positives. Hence, astandard deviation of the Gaussian function may be set to 3 (σ_(g)=3) toachieve both sensitivity against polyp edges and computationalefficiency, although other values are possible.

Edge Detection

Then, at process block 404, a crude set of edge pixels are detected inorder to characterize image appearance across polyp boundaries. To doso, any suitable edge detection method, such as a Canny's method, isapplied on the three color channels of the input images to extract asmany edges as possible. The goal is to obtain an overcomplete edge mapwhich will further be refined through a classification scheme.

Edge Direction Estimation

Next, at process block 406, oriented image patches are extracted alongthe edge direction. Particularly, as shown in FIG. 5A, the ground truthof a polyp region 500 is shown having corresponding upright imagepatches 502 and corresponding oriented image patches 504 for fourselected edges on a boundary 506 of the polyp region 500. The groundtruth is a binary image whose white pixels indicate the polyp region 500and black pixels represent the background region, as shown in FIG. 5A.While the polyp region 500 can appear in any portion of the uprightpatches 502, the polyp region 500 only appears on the right side of theoriented image patches 504.

Next, at process block 408, a robust and accurate estimation of edgenormal may be computed. The normal directions are used to extractoriented patches around the edges. In certain embodiments, an accurateestimate of edge normal is important to collecting sound orientedpatches 504. In some cases, edge normals computed using a gradient-basedapproach are often inaccurate, resulting in a non-smooth map of edgenormal and poorly aligned image patches. For example, as shown in FIG.5B, estimated edge normals for the region highlighted with the rectangle508 in FIG. 5A are shown using a gradient-based approach 510 and atensor voting approach 512. As shown in FIG. 5B, the gradient-basedapproach 510 yields an inaccurate estimation of edge normals, leading topoorly aligned oriented patches. However, the tensor voting approach 512operates very reliably except for edges lying on junctions, whosemisalignment effects can be mitigated with subsequent classification andvoting. The tensor voting approach 512 assumes that a ball tensor isplaced at each boundary pixel. In ball tensor voting, edge normal at apixel is determined according to the arrangement of the surrounding edgepixels such that the continuation of edge normal is maintained.

Feature Extraction and Classification

To capture the unique image appearance of polyps along the boundaries ofthe polyps, oriented sub-images, for example of size 64×64, may beextracted along the edge normals, as shown at process block 410. As aresult, in the extracted sub-images, the edges appear vertically in themiddle. For example, an edge pixel at angle θ, may include two possiblenormals (i.e., θ−π/2 and θ+π/2) that give two horizontally mirroredsub-images.

Patches, for example of size 8×16, are formed all over each sub-imagewith about 50% overlap along horizontal and vertical directions. Eachpatch may then be averaged vertically, resulting in a 1-dimensional (1D)intensity signal which presents intensity variation along the horizontalaxis. A 1D discrete cosine transform (DCT) may then be applied to obtaina compact and informative presentation of the signal. To achieveinvariance against constant illumination changes, the DC component(i.e., the average patch intensity) is discarded. To achieve invarianceagainst linear illumination scaling, the AC coefficients may be dividedby the norm of the DCT coefficients vector. Thus, the descriptor canpartially tolerate nonlinear illumination change over the wholesub-image, particularly if the nonlinear change can be decomposed to aset of linear illumination changes on the local patches. Then, the firstfew normalized AC coefficients may be selected from each patchcorresponding to low frequency intensity changes and concatenated toform a feature vector for a sub-image, as shown at process block 412.

The above described image descriptor provides rotation invariance, whichis important because features that can consistently represent imageappearance across edges lying at arbitrary directions are needed. Inaddition, the image descriptor provides illumination invariance, whichis important in colonoscopy since the source of light moves along withthe camera, thereby causing the same segment of a polyp boundary toappear with varying contrast in different frames.

In an alternative embodiment, image patches of polyp boundaries areobtained and oriented in order to construct principal component analysis(PCA) models that correspond to the R, G, and B color channels. To fullyutilize the information content of color patches, three PCA modelscorresponding to R, G, and B color channels may be constructed.Eigenvectors are then selected from each of the three PCA models suchthat approximately 90% of the total eigenvalue sum is covered, resultingin a 60-dimensional feature space. To generate appearance features, eachimage patch is projected along the selected eigenvectors and theresultant projection coefficients are concatenated to form the featurevectors. Compared to general purpose features, such as Haar andsteerable filters, PCA eigen images 600, as shown in FIG. 6, allow thepatterns of interest to be represented in a lower dimensional featurespace. More importantly, PCA naturally handles feature selectionaccording to the corresponding eigen values, which obviates the need foremploying sophisticated feature selection methods. The crude edge map isrefined by means of a classification model. More specifically, a randomforest classifier is used to filter out irrelevant, non-polyp edges(e.g., those edges lying on folds and vessels), while retaining edgesaround and on the polyp boundary.

Returning to FIG. 4, at process block 414, an edge classification isperformed to refine an edge map by filtering out “non-polyp” edges andto determine the normal direction for the retained polyp-like edges,such that the normals point toward polyp locations. To achieve theseobjectives, a two-stage classification system may be implemented wherethe first stage produces mid-level image features from low-level inputfeatures, while the second stage learns both edge label (i.e., “polyp or“non-polyp”) and normal direction. The classification system furtherincludes labeling of sub-negative classes, which can be done eithermanually or in an unsupervised way by applying Kmeans on negativepatches.

To train the first layer, N₁ oriented patches may be collected aroundboundaries of polyps and four sub-negative classes: vessels, lumenareas, specular reflections, and around edges at random locations intraining images. A five-class classifier may be trained using the patchdescriptor. The output of the classifier may be an array ofprobabilities for the five object classes. Compared with the low levelinput features, which encode local image variation, the generated outputarray contains mid-level features that measure the global similaritybetween the underlying input sub-image and the general appearance of thepredefined structures.

To train the second layer, N₂ pairs of oriented patches may be collectedfrom polyp boundaries and other random locations in training images. Inone non-limiting example, let {p_(i) ¹, p_(i) ²} be the extracted pairof patches around i^(th) edge with the corresponding normals, {n_(i) ¹,n_(i) ²}, where ∠n_(i) ¹ε[0,π) and ∠n_(i) ²=∠n_(i) ¹+π. Based on thestate of the i^(th) edge, a label, y_(i)ε{0, 1, 2} may be assigned toeach pair of patches, where “0” is for a non-polyp edge, “1” is for apolyp edge with normal being n_(i) ¹, and “2” is for a polyp edge withnormal being n_(i) ². Such labeling is possible given the ground truthfor polyps. Next, low level features may be extracted from each pair ofpatches and the classifier trained may be applied in the first layer,resulting in two arrays of mid-level features per pair that are furtherconcatenated to form a feature vector, {f_(i), y_(i)}. Once the featurevectors are collected, a three-class classifier may be trained to learnboth edge label and edge normal direction. In the test stage, the labelwith maximum probability may be assigned to the underlying edge pixel.

More generally, the second layer of classification fuses knowledgecaptured from a pair of patches because combining the two sources ofinformation can yield more accurate edge classification. For example, ifthe patch p_(i) ¹ around the i^(th) edge resembles the appearance of apolyp, and the counterpart patch p_(i) ² looks very similar to theaverage appearance of specular reflections or lumen areas, it may bedifficult to determine the underlying edge pixel. In one embodiment, thefirst patch may be relied on and a polyp edge with edge normal beingn_(i) ¹ may be declared. In another embodiment, information from thecounterpart patch may be considered and a non-polyp edge may bedeclared. In yet another embodiment, to determine the underlying pixel,a second classifier in the mid-level feature space may be trained tofully utilize such relationships.

Vote Accumulation for Polyp Localization

At the next process block 416, a polyp vote accumulation analysis isperformed. In the ideal classification scenario, all non-polyp edgepixels are removed and the arrangement of positive pixels indicates thelocations of polyps. However, in practice, a portion of non-polyp edgesmay pass the classification stage and induce false positives. On theknowledge that false positive edges often appear on elongated andlow-curvature edge segments, a vote accumulator scheme that willmitigate the effect of false positive edges, but also enable polyplocalization from fragmented edges, may be utilized. The voteaccumulation scheme assigns high values to the regions that arepartially surrounded by curvy edges, but gives low values to the regionsthat are surrounded by elongated low curvature edges.

Turning to FIG. 7A, in a vote accumulator scheme, the edges that havepassed the classification stage may be referred to as “voters.” Eachvoter includes a polyp direction n_(i)* and a classification confidenceC_(vi)=max(p(y^(i)=1), p(y^(i)=2)). The voting scheme begins withgrouping the voters into K categories according to the voting directionsof the voters,

${V^{k} = \left\{ v_{i} \middle| {\frac{k\; \pi}{K} < {\angle \; n_{i}^{*}} < \frac{\left( {k + 1} \right)\pi}{K}} \right\}},{k = {0\ldots \; {K.}}}$

Such edge grouping prior to vote casting minimizes vote accumulation inthe regions that are surrounded by low curvature boundaries. The votersin each category then cast votes at their surrounding pixels accordingto their voting directions and classifications confidence. This resultsin K voting maps that are further multiplied to form the final votingmap whose maximum vote accumulation (MVA) indicates the location of apolyp candidate, as shown the equation below:

$\begin{matrix}{{{MVA} = {\begin{matrix}{\arg \mspace{14mu} \max} \\{x,y}\end{matrix}\mspace{14mu} \Pi_{k = 1}^{K}\Sigma_{v \in V_{k}}\mspace{14mu} {M_{v}\left( {x,y} \right)}}},} & {{Eqn}.\mspace{14mu} (1)}\end{matrix}$

where M_(v)(x, y) is the vote cast by the voter v at a receiver pixelr=[x, y], which is computed as follows:

$\begin{matrix}{{M_{v}(r)} = \left\{ \begin{matrix}{{C_{v}\mspace{14mu} {\exp \left( \frac{- \left. ||\overset{\rightarrow}{vr} \right.||^{2}}{\sigma} \right)}\mspace{14mu} {\cos \left( {\angle \overset{\rightarrow}{n^{*}}\overset{\rightarrow}{vr}} \right)}},} & {{{if}\mspace{14mu} \angle \overset{\rightarrow}{n^{*}}\overset{\rightarrow}{vr}} < {\pi \text{/}2}} \\{{0,}\mspace{304mu}} & {{{if}\mspace{14mu} \angle \overset{\rightarrow}{n^{*}}\overset{\rightarrow}{vr}} \geq {\pi \text{/}2}}\end{matrix} \right.} & {{Eqn}.\mspace{14mu} (2)}\end{matrix}$

where σ controls the size of the voting field. FIG. 7A shows the votingfield for an edge pixel lying at 135 degree. As shown, the votes arecast in the region pointed by the voting direction. Such selectivityarises from the condition set on ∠{right arrow over (n*)}{right arrowover (vr)}, which inhibits the voters from casting votes in the oppositedirection. The exponential and cosinusoidal decay functions enablesmooth vote propagation, which may be used to determine the likelihoodof a polyp candidate.

However, in the accumulation scheme described above, votes received ateach pixel are accumulated irrespective of the voters' orientation. Thisimplies that the proposed accumulator may be undesirably sensitive toany accumulation of votes, no matter whether they are received from theedge pixels forming a circle or from the edge pixels arranged onparallel lines. Thus, regions delineated by parallel edge segments or bylow curvature counters, in general, may not represent polyps. Highresponses in such regions may result in false positive detections.

As shown in FIG. 7B, the voting map of the edge pixels arranged on acircle 700 is compared to the voting map of the edge pixels lying on twoparallel lines 702. Thus, votes are accumulated in two regions, namely,inside the circular edges which is desirable, and between the parallellines which is undesirable.

To overcome this problem, a constraint may be imposed on the votingscheme, such that regions can attain high accumulated votes if theyreceive adequate votes from voters whose normal directions span a widerange of [0, 2π), for example. Thus, regions surrounded by low curvatureboundary receive low accumulation, because the edge normal of theirsurrounding voters can only partially cover the range of [0, 2π).Despite the proper handling of low curvature boundaries, this constraintundesirably weakens vote accumulation for the polyps that partiallyappear behind the folds or close to image borders. In such cases, polypsare segmented incompletely and, thus, the angular range of interest isonly covered partially.

To alleviate the above problem, normal directions from [0, 2π) to [0,π), for example, can be mapped and a constraint can be placed on itsmaximum coverage. This allows for detecting semi-elliptical structuresin edge maps. To measure the coverage, normal directions can bediscretized into four values, for example. Mathematically, this may beexpressed as:

$\begin{matrix}{\theta^{i} = {\left. {i\frac{\pi}{8}} \middle| i \right. = \left\{ {1,3,5,7} \right\}}} & {{Eqn}.\mspace{14mu} (3)}\end{matrix}$

In one non-limiting example, the edge normal in the ranger of

$\left( {0,\frac{\pi}{4}} \right\rbrack$

is mapped to

$\frac{\pi}{8},$

the edge normal in the range of

$\left( {\frac{\pi}{4},\frac{\pi}{2}} \right\rbrack$

is mapped to

$\frac{3\pi}{8},$

and so on. The positively classified edges may then be categorized intofour categories according to the quantized values, and the votingprocess can be performed four times. Each time the voting process isperformed, the edge pixels of one specific category are allowed to vote.As illustrated by process block 310 of FIG. 3, the resultant number of,for example, four, voting maps are later multiplied to generate a finalvoting map.

Turning now to FIG. 7C, the effect of the modified voting map for thesame edge pixels is shown. The modified scheme assigns low values to theregion between the parallel lines, and high values to the region insidethe circular edges. The parameter of the voting scheme is the size ofvoting field controlled by σ_(v), which can be automatically adjustedgiven the size of the image and the upper and lower bounds onpolyp/image size ratio.

Returning to FIG. 4, a post polyp voting mechanism may be performed atprocess block 418. To assign a probability to a polyp candidate, a rayback projection may be performed in which radial rays are cast from thedetection location outward in all possible directions and then thefraction of rays hitting the voters is calculated. The ray backprojection may be used to determine the search radius for eachindividual ray. Short or long radii may underestimate or overestimatethe polyp likelihood. Search radius may be estimated by modeling thedecay in vote accumulation along each radial ray. For a ray at angle θ,the search radius is estimated as 3σ_(θ) where σ_(θ) is the standarddeviation of the Gaussian function fitted to the corresponding decaysignal. As shown in FIG. 7D the outwardly extending lines representsearch radii 704 for a subset of radial rays. Once search radii 704 aredetermined, the probability of a polyp candidate is measured as

${\frac{1}{180}{\Sigma_{\theta = 0}^{179}\left( {R_{\theta}\mspace{14mu} V\mspace{14mu} R_{\theta + 180}} \right)}};$

where R_(θ) is an indicator variable that takes 1 if the ray at angle θhits at least 1 voter, and 0 otherwise.

In an alternative embodiment, the post polyp voting mechanism performedat process block 418 may include defining isocontours of a voting mapand then using the isocontours to estimate the unknown parameters of thebands. The isocontour Φ_(c) of the voting map V may be defined asΦ_(c)={(x, y)|V(x, y)=cM} where M denotes the maximum of the voting mapand c is a constant between 0 and 1. The isocontours of a voting map canpredict where actual object boundary is through a regression model.Since isocontours may get corrupted by other nearby voters in the scene,several isocontours may be obtained, as shown in FIG. 8A, and then taketheir median shape as the representative isocontour Φ of the voting map,as shown in FIG. 8B. Let d_(iso) ^(i) denotes the distance between thei^(th) point on Φ and MVA. d_(iso) ^(i) may be used to predict d_(obj)^(i), the distance between the corresponding point on the objectboundary and the MVA within a prediction interval. For this purpose, asecond order polynomial regression model d_(obj) ^(i)=b₀+b₁(d_(iso)^(θ))+b₂(d_(iso) ^(i))² may be used, where b₀, b₁, and b₂ are theregression coefficients and are estimated using a least square approach.Once the model is constructed, the output of the model d_(obj) is takenat angle θ with respect to MVA as t_(θ) and the corresponding predictioninterval as the bandwidth δ. With this information, the band around thepolyp candidate can be formed and the probability may be computed.

Referring again to FIG. 2, at decision block 206, if a polyp is notdetected, the above process may be repeated, wherein a next opticalimage is acquired at process block 202, and so on, as desired or asadditional optical images are available. However, if a polyp ispositively identified, then, at process block 208, an alert or signalmay be provided to an operator, indicating a positive polypidentification. The alert or signal may take any suitable shape or form,such as an audible or visual cue. At decision block 210, if all theimages have been processed (i.e., the colonoscopy procedure iscomplete), a report is generated at process block 212, which may takeany suitable shape or form. However, if all the images have not beenprocessed at decision block 210, additional optical images are acquiredand processed at process block 202 to determine if more positive imagescan be found.

Specific examples are provided below, illustrative of theabove-described polyp detection method. These examples are offered forillustrative purposes only, and are not intended to limit the scope ofthe present disclosure in any way. Indeed, various modifications of thedisclosure in addition to those shown and described herein will becomeapparent to those skilled in the art from the foregoing description andthe following example and fall within the scope of the appended claims.

Example

As a non-limiting example, a CVC-ColonDB was used to evaluate themethodology described in the current disclosure, where CVC-ColonDB isthe only publicly available polyp database, consisting of 300colonoscopy images with 20 pedunculated polyps, 180 sessile polyps, and100 flat polyps. The patch descriptor was evaluated first and then thewhole polyp detection system was compared with the state-of-the-artmethods.

Feature Evaluation

For feature evaluation, 50,000 oriented images around polyp and otherboundaries in colonoscopy images were collected. Half of the imagescorresponding to the first 150 images were selected for training arandom forest classifier and used the rest for testing. FIG. 9A comparesthe resultant ROC curves of the current patch descriptor using threeselected coefficients (i.e., 315 features per sub-image) and thoseobtained from other widely-used methods, such as HoG¹, LBP², and Daisy³.Due to space constraints, the ROC curves corresponding to more than 3DCT coefficients were excluded, however the experiments demonstratedthat it did not achieve any significant improvement in performance. Interms of running time, the current patch descriptor outperformed theDaisy descriptor. FIG. 9B shows the discrimination power of each featureextracted by the current patch descriptor across all 50,000 images.Comparing the discrimination map against the average appearance of polypboundary, as shown in FIG. 9C, reveals that the most informativefeatures are extracted from the polyp boundary, indicating that thepresent descriptor successfully captures the desired image information.FIG. 9D shows the ROC curve for the present polyp detection system.

Voting Scheme and Polyp Detection

For system evaluation, a five-fold cross validation was implemented. Totrain the two-stage classifier, N₁=100,000 oriented image patches werecollected from training images with approximately 20,000 samples foreach of the five predefined classes, and N₂=100,000 oriented imagepatches were collected with 50% of images being extracted around polypsand the rest around edges in random location in training images. Forclassification, the random forest classifier was chosen because of itshigh quality probabilistic output that is utilized in the voting scheme.The trained classification system, followed by the voting scheme, wasthen applied to the five test folds. The voting scheme detected 267 outof 300 polyps. Thus, outperforming the state-of-the-art, where only 252candidates were cast inside the polyps.

Polyps appear in different sizes in colonoscopy images. Because largepolyps can often be effortlessly detected by colonoscopists, σ_(v) maybe adjusted for detecting polyps of small and moderate sizes.Considering that the area of missed polyps are usually 9 to 16 timessmaller than images, σ_(v) may vary between 70 and 90. A detection maybe considered as a “true detection” if the maximum of the voting mapfalls inside the ground truth contour provided in the database. Theaccuracy of the polyp detection method can be measured by adding thetrue and false detections generated in the five test folds. Forσ_(v)ε[70, 90], the detection point was placed inside 267 polyps andproduced 33 false detections which outperforms SA-DOVA descriptor with252 true and 48 false detections (p<0.05, Z=2.01). Furthermore, whileSA-DOVA descriptor requires five parameters to be tuned, the presentdisclosure has only two parameters, namely, σ_(v) whose range ofvariation can be automatically set, and σ_(g), which as shown in FIG. 10gives comparable performance with changes between 3 and 7.

Examples of polyp localization are shown in FIG. 11. The voting maps areshown superimposed on the colonoscopy images 900 for bettervisualization. For each image, the detection point 902 is the maximum ofthe voting map. The black contour shows the ground truth, and the greenpixels 904 are the edges that have passed the classification stage. Thefalse detections mostly occurred in images with no clear boundarybetween polyps and their surrounding area.

To obtain precision and recall rates, a threshold was changed on theprobabilities assigned to the generated polyp candidates. A non-limitingexemplary experiment was conducted to compare one embodiment of thesystem of the present disclosure with conventional systems. As shown inthe table of FIG. 12, the proposed system outperformed previous systemswhere Haar features, a one-layer classification, and a less accuratevoting scheme were employed. Furthermore, the system was evaluated byincluding 3,000 images that did not contain a polyp (i.e., 300 positiveimages from CVC-ColonDB and 3000 negative images from a privatedatabase).

In a different system evaluation, a five-fold cross validation wasimplemented to evaluate the polyp detection system using CVC-ColonDB. Adetection was “true” if it fell inside the ground truth. As shown in thetable of FIG. 13, the proposed system outperformed previous systems indifferent operating points and yields stable results over a relativelylarge range of a for precision and recall rates.

Examples of polyp localization are shown in FIG. 14. The voting maps areshown superimposed on the colonoscopy images 1000 for bettervisualization. For each image, the detection point 1002 is the maximumof the voting map. The black contour shows the ground truth, and thegreen pixels 1004 are the edges that were retained after theclassification stage. Line segments 1006 that reach polyp edges withdesired voting directions, are also shown in FIG. 14. False candidatesproduced by the voting scheme mostly occur due to aggressive edgeclassification and inadequate clear boundary between polyps and theirsurrounding area.

The system trained on the entire CVC-ColonDB was further evaluated usingeight short colonoscopy videos. The free-response ROC curve, as shown inFIG. 15, demonstrates that the suggested two-stage classification schemesignificantly outperforms a one-stage classification scenario where athree class classifier is used for edge classification.

In summary, colorectal cancer most often begins as abnormal growth ofthe colon wall, commonly referred to as polyps. It has been shown thatthe timely removal of polyps with optical colonoscopy can reduce theincidence and mortality of colorectal cancer. However, polyp detectionwith optical colonoscopy is a challenging task and as reported, manypolyps remain undetected. Computer-aided detection may offer promises ofreducing polyp miss-rate.

The current disclosure describes a system and method that systematicallyexploits the unique appearance of polyp boundaries to suppress non-polypedges, yielding a cleaner edge map for the above-described voteaccumulation scheme. This approach can accommodate a large variation ofpolyp shapes, eliminate parallel edge configurations, and enable polypdetection from partially identified boundaries of polyps. Thus, thesystem and methods disclosed can improve the classification stage byenhancing the feature space with classification systems trained inconsistent subspaces of the negative class (i.e., the boundaries oflumen, vessels, and folds, etc.), as well as evaluate the suggestedmethodology on a significantly larger polyp database.

In addition, the present polyp detection system and method providesfeedback to enhance colonoscopists' diagnostic capabilities, especiallyduring long and back-to-back colonoscopies, where human factors, such asinsufficient attentiveness and fatigue, result in misdetection ofpolyps. In addition, the current disclosure describes a method that isnot limited to optical colonoscopy and have provided effectiveness forpolyp detection in capsule endoscopy.

The present disclosure has been described in terms of one or moreexemplary embodiments, and it should be appreciated that manyequivalents, alternatives, variations, and modifications, aside fromthose expressly stated, are possible and within the scope of thedisclosure.

We claim:
 1. A system for automated polyp detection in opticalcolonoscopy images, the system comprising: an input configured toacquire a series of optical images; a processor in communication withthe input; and a memory containing instructions that, when executed bythe processor, causes the processor to perform a process on the seriesof optical images comprising: applying a color filter to create aplurality of color filtered images for each optical image of the seriesof optical images; locating a series of edge pixels on at least one ofthe plurality of color filtered images; obtaining a plurality oforiented image patches corresponding to each of the series of the edgepixels, the plurality of oriented image patches represented by anintensity signal characterized by at least one of rotation invariance orillumination invariance; estimating an edge normal for each edge pixel;constructing at least one classification system corresponding to theplurality of color filtered images, the at least one classificationsystem configured to enhance low level features of the plurality ofcolor filtered images prior to classification; generating appearancefeatures from a series of feature vectors selected from the at least oneclassification system; performing an edge classification thresholdanalysis on each of the plurality of oriented image patches;determining, based on the edge classification threshold analysis, that agiven image patch is consistent with an edge threshold; and generating areport indicating potential polyps with a vote accumulation greater thana threshold, the vote accumulation including a probabilistic output foreach of the potential polyps in the absence of a predefined parametricmodel.
 2. The system of claim 1, wherein the color filter includes a redfilter, a green filter, and a blue filter.
 3. The system of claim 1,further comprising a Canny edge detection algorithm stored on the memoryand applied to the plurality of color filter images when locating theseries of edge pixels on at least one of the plurality of color filteredimages.
 4. The system of claim 1, wherein the edge normal for each ofthe series of the edge pixels is estimated by a tensor voting with aball tensor placed at a location of each of the series of edge pixels.5. The system of claim 1, wherein each of the series of edge pixels areassigned at least one classification score based on an estimated edgenormal, wherein when the at least one classification score is less than0.5, a corresponding edge pixel of the series of edge pixels iseliminated.
 6. The system of claim 1, wherein the process furthercomprises comparing the plurality of oriented image patches and theseries of feature vectors when performing the edge classificationthreshold analysis on each of the plurality of oriented image patches.7. The system of claim 1, wherein the process further comprises applyinga random forest classifier, whereby the edge threshold is in a range of0.5 to 1.0 when determining, based on the edge classification thresholdanalysis, that the given image patch is consistent with the edgethreshold.
 8. The system of claim 1, wherein for each edge pixel in eachoptical image, the vote accumulation is performed for each of one ormore categories to generate a plurality of voting maps.
 9. The system ofclaim 8, wherein the at least one classification system includes atwo-stage classification system to filter out non-polyp edges and retainpolyp edges around and on a boundary of the polyp.
 10. The system ofclaim 1, wherein the vote accumulation indicates an output for eachpolyp in absence of a parametric model of polyps by detecting curvyboundaries along the series of edge pixels.
 11. A method for automatedpolyp detection in optical colonoscopy images, the method comprising:applying a color filter, using a processor in communication with aninput configured to acquire a plurality of optical images, to create aplurality of color filtered images for each of the plurality of opticalimages; locating a series of edge pixels on at least one of theplurality of color filtered images; obtaining a plurality of orientedimage patches corresponding to each of the series of the edge pixels,the plurality of oriented image patches represented by an intensitysignal characterized by at least one of rotation invariance orillumination invariance; estimating an edge normal for each edge pixel;constructing at least one classification system corresponding to theplurality of color filtered images, the at least one classificationsystem configured to enhance low level features of the plurality ofcolor filtered images prior to classification; generating appearancefeatures from a series of feature vectors selected from the at leastclassification system; performing an edge classification thresholdanalysis on each of the plurality of oriented image patches;determining, based on the edge classification threshold analysis, that agiven image patch is consistent with an edge threshold; and generating areport indicating potential polyps with a vote accumulation greater thana threshold, the vote accumulation including a probabilistic output foreach of the potential polyps in the absence of a predefined parametricmodel.
 12. The method of claim 11, wherein the color filter includes ared filter, a green filter, and a blue filter.
 13. The method of claim11, wherein locating the series of edge pixels on the at least one ofthe plurality of color filtered images comprises applying a Canny edgedetection algorithm to the plurality of color filter images.
 14. Themethod of claim 11, further comprising estimating the edge normal foreach of the series of the edge pixels by a tensor voting with a balltensor placed at a location of each of the series of edge pixels. 15.The method of claim 11, further comprising assigning at least oneclassification score to each of the series of edge pixels based on anestimated edge normal, wherein when the at least one classificationscore is less than 0.5, a corresponding edge pixel of the series of edgepixels is eliminated.
 16. The method of claim 11, further comprisingcomparing the plurality of oriented image patches and the series offeature vectors when performing the edge classification thresholdanalysis on each of the plurality of oriented image patches.
 17. Themethod of claim 11, further comprising applying a random forestclassifier, whereby the edge threshold is in a range of 0.5 to 1.0 whendetermining, based on the edge classification threshold analysis, thatthe given image patch is consistent with the edge threshold.
 18. Themethod of claim 11, further comprising performing the vote accumulationfor each edge pixel in each optical image for each of one or morecategories to generate a plurality of voting maps.
 19. The method ofclaim 18, wherein the at least one classification system includes atwo-stage classification system to filter out non-polyp edges and retainpolyp edges around and on a boundary of the polyp.
 20. The method ofclaim 11, wherein the vote accumulation indicates an output for eachpolyp in absence of a parametric model of polyps by detecting curvyboundaries along the series of edge pixels.